
m 



CALL- FLOW VERIFICATION METHOD AND APPARATUS 



INVENTOR: Thomas S. Neary 



FIELD OF THE INVENTION 



This invention relates to testing of interactive 
audio systems and, more particularly, to verification of 



voice response system in the course of processing a user 



Banks, brokerages and other businesses now enable a 
customer to obtain account information by calling an 
automated telephone system. These systems, which will 
be referred to by the general term interactive voice 
response system or IVR system, generate computer- 
provided pre-recorded, synthesized or re-constituted 
"voice" messages to guide a caller in entering 
identification and inquiry data via telephone keypad 
activation, by voice input, or both. For example, after 
initiating a call the caller may hear an audio message 
(which may for present purposes be referred to as an 
utterance) requesting entry of the caller's account 
number, followed by a further series of utterances 
appropriate to process the caller's inquiry, 

A problem separate and apart from the technical 
implementation of such systems is the need to provide 
verification of accuracy of operation on an initial and 
continuing basis, particularly after program 
modification. Verifying the content and flow of 
utterances provided during the course of a single call 



content and flow of messages or prompts provided by a 



call. 



BACKGROUND OF THE INVENTION 
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to an IVR system is relatively straight forward when 
performed manually. Accurate processing of a larger 
number of calls proceeding simultaneously or in close- 
time contiguity in an IVR system is correspondingly 
5 complex. Capabilities are needed to enable operators of 
IVR systems to verify accuracy of performance. 



SUMMARY OF THE INVENTION 



The present invention enables automated 
verification of utterance content and call flow 
10 performance by an interactive audio system. In a call 
connection to such a system for call-flow verification 
purposes, appropriate data inputs are provided to 
"'"-i simulate a typical customer inquiry and data entry. To 

i;n enable verification processing, the interactive audio 

'r: 15 system incorporates the capability of providing coded 
signals representative of content of each utterance in 
coded format for inclusion in prompt signals. By 
comparing the content of utterance labels, as 
,£ represented by such coded signals transmitted in the 

20 course of a call connection, to pre-stored data 

representing the content of correct utterance labels for 
the interactive audio system under test, discrepancies 
can be identified and reported. Such discrepancies may 
represent, for example, utterance content errors or 
25 omissions, call flow errors or omissions, or 
combinations thereof . 

For purposes of call-flow verification testing, the 
interactive audio system will, for example, be operable 
in a unique call -flow verification mode (termed a CFV 
30 mode) and such CFV mode may be selectable by inclusion 
of a CFV sequence code in a call connection to the 
interactive audio system. Pursuant to the invention, a 
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CFV sequence code may typically include coded 
instructions to activate the CFV mode, to include or 
exclude actual utterances with the coded representations 
thereof, and to define the extent to which utterance 
5 label content is to be represented in coded signal 
format (e.g., initial and end portions, or all 
characters of an utterance label) . By enabling 
automated call-flow verification for a single call 
connection, the invention enables automated verification 

10 of larger numbers of calls since each call is 

independently processed and results may be stored for 
analysis. Automated verification may be carried out on 
a call by call basis, without interfering with actual 
customer calls which can be intermixed. 

15 For a better understanding of the invention, 

together with other and further objects, reference is 
made to the accompanying drawings and the scope of the 
invention will be pointed out in the accompanying 
claims. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows a call -flow verification system. 

Fig. 2 illustrates data entry and responsive 
utterances during a simulated user call session. 

Fig. 3 shows an interactive audio system having a 
25 call-flow verification system and set-up for call -flow 
verification . 

Figs. 4, 5 and 6 illustrate examples of prompt 
signals including content of utterance labels encoded in 
DTMF format . 

30 Fig. 7 illustrates encoding of utterance labels in 

prompt signals in the course of a simulated user call 
session during call-flow verification. 
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Fig. 8 is a flow chart useful in describing call- 
flow verification methods. 

Fig. 9 illustrates a call-flow verification (CFV) 
sequence code for activating the CFV mode of an 
interactive audio system. 



DETAILED DESCRIPTION 



Fig. 1 illustrates an equipment configuration 
suitable for use of a call-flow verification method. In 
this example, the three IVR systems 10, 11, 12 are used 

10 to provide a high-volume call handling capacity for a 
brokerage or other business application. Each of 
systems 10, 11, 12 may be provided by Periphonics VPS 
systems as commercially available from Nortel Networks 
of Bohemia, New York. As shown, IVR systems 10, 11, 12 

15 are arranged to receive incoming calls via network 14, 
which may be a public or private telephone network, 
Internet or direct connect facility, or combination of 
these or other suitable communicative coupling 
facilities. For example, an IVR system can be deployed 

20 in a customer care center infrastructure, such as an 
Automatic Call Distribution ( ''ACD" ) center. Further 
detail is provided in U.S. Patent No. 5,987,115, issued 
November 16, 1999, to Petrunka et al . , which is 
incorporated herein by reference. Customer care center 

25 infrastructures are also available as a Symposium™ Call 
Center product from Nortel Networks, Inc., of Ontario, 
Canada . 

Fig. 1 includes automated call generator (ACQ) 
units 16 17, 18 arranged to place calls to IVR systems 
30 10, 11, 12 via network 14. As will be further 

described, ACG units 16, 17, 18 may be provided by 
Periphonics VPS systems programmed to automatically 
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place calls which simulate customer calls, including 
voice and dual -tone multi- frequency (DTMF) signals 
representing customer speech and telephone keypad 
actuations. By inclusion of three ACG units 16, 17, 18, 
5 a large volume of independent calls can be placed 

simultaneously or in short time periods to simulate peak 
volume customer activity. 

The Fig . 1 equipment also includes a server 
facility 20 and a control console 22. Server 20 
10 includes suitable software and storage capabilities to 
provide a database management system with processing, 
storage and access of and to call scripts, IVR responses 
and other data and information as will be further 
=.3 (described. Call scripts that exercise all call flow 

15 paths (e.g., all go-right and error paths in an IVR 
^ I application) , including simulated user voice and keypad 

inputs, can thus be stored for use and repetitive re-use 
by the ACG units. Generally, a script is understood to 
be a type of computer code that can be directly executed 
\2 20 by a program that understands the language in which the 
;M script is written. Scripts do not require compilation 

Q into object code to be executed. Console 22 may be a 

computer work station executing software suitable for 
control and management of the equipment and its 
25 operation, as will be further described. Server 20 may 
be a Periphonics Relational Database Management System 
(RDBMS) , and console 22 may be a VPS Console, both as 
commercially-available from Nortel Networks. 
/ For purposes of ease of understanding, attention 

30 / will be directed to a single call (simulating a series 
\vOf interchanges during the course of one customer call) 
/placed to IVR system 10 by ACG unit 16 over a network 

_ \l^. Not e ho wever, that^ a large number of such calls of 

identical or different content may be placed 
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10 



15 



20 



25 



30 



sequentially or in overlapping time periods. 

As a basic overview, an example of a call operation 
is as follows. IVR system 10 is programmed to respond 
to an in-bound call by greeting the originating caller 
and then, via the same in-bound call connection, 
transmitting an utterance from the destination, such as 
"For account balance, press one. To open an account, 
press two." Upon the originating caller responding by 
activating the "one" key of the terminal keypad, a 
correct utterance by IVR system 10 might be "Please 
enter your three-digit account number." The originating 
caller would then respond by activating three keys of 
his terminal keypad. These and successive user inputs 
and IVR system utterances are illustrated in Fig. 2, 
which represents the basic telephony interchanges of a 
demonstration session for call-flow verification (not 
including coded utterance content, which will be further 
described below) . 

As simulated calls are placed and progress, at 
least three categories of data storage are involved for 
purposes of call-flow verification. First, the IVR 
system 10 is pre-programmed by storage (e.g., directly 
or via speech reconstruction or synthesis techniques) of 
specific utterances which are intended to be appropriate 
in processing and responding to a user call and user 
responses received in the course of that call. Examples 
of such utterances are provided in the "IVR ACTION" 
^column of Fig. 2. Second, the ACG unit 16 is pre- 
programmed for call-flow verification of this particular 
IVR system 10 by storage of data_ re presen ta tive of 
content of the same specific utterances_ which IVR system 



j 10 is intended to provide. For this purpose, the actual 
utterances or data representative of the content of such 



utterances may be stored to be available for access by 
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ACG unit 16, for example. Third, provision is made for 
receiving and storing the actual utterances, or data 
representative of the content of such utterances, as 
actually issued by IVR system 10 in the course of 
responding to a simulated user call. As will be • 
described further, the second and third categories of 
data storage enable comparison of content of actual 
utterances with content of correct utterances, to enable 
identification of discrepancies and thereby verification 
l^of call-flow performance by IVR system 10. 

To enable call-flow verification, IVR unit 10 has 
call-flow verification (CFV) apparatus including 
circuits 30 and 32 shown in Fig. 3. Circuit 30 is an 
encoding circuit utilized to provide coded signals 
representative of content of utterances in coded format 
for inclusion in prompt signals sent by IVR system 10. 
Circuit 32 is an activation circuit to control 
activation and deactivation of encoding circuit 30. 
Activation circuit 32 also controls whether, during 
/call-flow verification, prompt signals sent by IVR 
/ system 10 include or exclude the actual utterances whose 
\pontent is represented by the coded signals, as well as 
how many characters of an utterance label are 
represented by the coded signals. 

Capabilities of ACG unit 16 used for call-flow 
verification processing include circuits 34 and 36 shown 
in Fig. 3. Circuit 34 is a decoding circuit utilized to 
decode the coded signals representative of content of 
utterances. As will be further described, circuit 34 is 
effective to recover characters of utterance labels in 
ASCII format, from received signals encoded in DTMF 
format, for example. Circuit 36, as shown, is a 
compar i son circui t ^ to compare characters of received 
utterance labels with previously stored characters of 
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correct utterance labels, as will be further described. 
Implementation of circuits 30, 32, 34 and 36 in 
appropriate form can be carried out by skilled persons 
using established techniques. 

Reference is made herein to storage of data 
representative of the content of utterances and to coded 
signals representative of such content. The data may be 
stored in any appropriate format . As to such coded 
signals, in a currently preferred embodiment coded 
10 signals representative of utterance content are provided 
in the form of utterance labels, encoded in DTMF format. 
Thus, internally to IVR system 10, each utterance (e.g., 
a voice or audio message) is identified for retrieval 
and other purposes by an utterance label (e.g., in ASCII 
15 format, as distinct from the actual audio for an 

utterance) . Each such label may be an assigned word, 
phrase or alphanumeric series, may be a portion or all 
of the utterance expressed in ASCII format, or may be 
other suitable form of label . For purposes of 
20 transmissions between units 10 and 16, for example, an 
utterance may be represented by coded signals in DTMF, 
multi-frame (MF) or other suitable format, as will be 
further described. For exemplary purposes in the 
present description, it will be assumed that each 
25 utterance label is a representation of all characters of 
an utterance, expressed in ASCII format. 

In a presently preferred embodiment, some or all of 
the alphanumeric characters of an utterance label are 
represented by coded signals in the form of DTMF 
30 signals. The CFV device in IVR system 10 is arranged so 
that during call-flow verification the format of prompt 
signals sent by IVR system 10 is determined by encoding 
circ uit 30, under the control of activation circuit 32. 



That is to say, the format as to the inclusion of coded 
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signals and inclusion or exclusion of the utterances 
whose content is represented by such coded signals, is 
determined by encoding circuit 30, in effect, overriding 
the customary operation of IVR system 10 as provided in 
the absence of call-flow verification activation. Also, 
it is arranged so that activation circuit 32 is 
responsive to a CFV sequence code sent to IVR system 10, 
"or example by DTMF signals representing the CFV 
sequence which are received via an incoming call, direct 
connection, etc., to activate CFV for a series of calls 
or on a per call basis. 

Operation of encoding circuit 30 will be further 
considered. As noted, coded signal s can be provided iiT 
the form of DT jjjjFL^siQ nal^ alphanumeric 
characters of an utterance label in ASCII format. 

■ . ■■ : 

At least three basic parameters are involved in 
operation of encoding circuit 30: an on/off parameter, a 
framing parameter and an extent parameter. 

First, with the on/off parameter circuit 3 0 can be 
either activated, for call-flow verification, or 
deactivated, permitting operation of IVR system 10 for 
processing of customer calls during the ordinary course 
of business. It will be appreciated that if encoding 
circuit 3 0 were activated during processing of customer 
calls, DTMF signals included for call -flow verification 
purposes would be audible to the customer in this 
embodiment. In other embodiments coded signals in 
subliminal or other format which are inaudible or not 
noticeable by a caller may be employed. 

Second, the framing parameter, with inclusion of 
coded signals (representing content of a specific 
utterance) in prompt signals to be sent cby/IVR s;i^stem 



J.0, the ac tual utterance it self may be inc lude d or 
excluded. Since the coded signals alone may be used for 



present call-flow verification purposes, it is typically 
not necessary to include the utterance itself. However, 
the audio ut terance (e.g., in the form of audible 
speech) may be included so that after reception and 
storage it will be available for analysis as to clarity, 
sound level, etc. 

I Third, the extent parameter, in representing the 

content of an utterance the coded signals can, for 

example, represent the first few characters and the last 

few characters of an utterance label or, alternatively, 

each character of the entire utterance label. 

These parameters, as referred to above in respect 
^ . ' ^ — -p^ 

to operation of encoding .circuit 30, will be further 
illustrated by consideration of the examples provided in 
Figs. 4, 5 and 6. The first, or on/off activation 
parameter is set for "on" (i.e., activation of call-flow 
verification) in these examples. 

Fig. 4 illustrates prompt signals including coded 
signals (DTMF signals) for the one -wo rd u tterance "one" . 
In this example, the secjond_parameter , or the "frjijningj' 
garajneter, is set for inclusion of the (a^tual_ utte ranee .) 
The third parametjgr, or the "extent" parameter, is set 
for five characters. In this example, as illustrated in 
Fig. 4, for call-flow verification the prompt signals 
transmitted by IVR system 10 consist of the actual 
utterance "one" preceded by DTMF signals representing 
the letters "o" , "n" , "e" in ASCII format (i.e., 
representing the decimal equivalents thereof) and 
followed by the same DTMF signal content again 
representing each letter of the utterance label "one" . 
In the present coding protocol, setting the extent 
parameter to "5" provides for encoding of the first^jive 
ch ara cters of an utt era nce^ lab^l^j^ a_J._ead ing g g^^^^^gg of 
DTMF signals and the last five characters in a trailing 
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series of DTMF signals. Here, since the utterance label 
includes only three characters, all three characters are 
represented in both the leading and trailing coding. 
Fig. 5 illustrates prompt signals provided for 
5 call-flow verification regarding the utterance "Press 

one for date" . Again, the framing parameter is set for 
inclusion of the actual utterance and the extent 
parameter is set for five characters. As shown, the 
leading series of DTMF signals represents the five 
10 letters of "press" and the trailing DTMF signals 

represent the five characters "space", "d" , ""a" , "t" , 
"e" , For this purpose, the space between words is 
treated as a character and represented by appropriate 
ASCII coding. 

15 Fig. 6 also addresses the utterance "Press one for 

date" . However, in this example the second or framing 
parameter is set for exclusion of the actual utterance 
and the third or extent parameter is set for "all" 
characters. Thus, the actual utterance is not included 

20 in the prompt signals, but the eighteen characters 
(including spaces) of the utterance label are 
represented by a series of DTMF signals representing the 
characters in ASCII format. 

Fig. 7 provides a simplified example of 

25 transmissions between ACG unit 16 and IVR system 10 

during the course of a simulated user call for call-flow 
verification purposes. To the left in Fig. 7 there is 
represented performance of an application program 
executed on IVR system 10 and programmed to provide 

30 utterances responsive to the user call. In operation of 
the application program the appropriate utterances are 
identified for retrieval, synthesis, etc., by utterance 
labels as discussed above. These utterance labels are 
thus available for use in encoding selected characters 
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or all characters of a specific utterance label in coded 
form for inclusion in a prompt signal transmitted by VPS 
system 10. 

As will be seen, starting at the top of Fig. 7 and 
proceeding item by item downward, ACG unit 16 first 
dials a call to IVR system 10 and then provides a CFV 
sequence code (in the form of DTMF signals) effective in 
this example (i) to activate the CFV mode of operation 
of IVR system 10, (ii) to indicate that the actual 
utterances are to be included in the prompt signals and 
(iii) to set the extent parameter to two characters. 
Accordingly, after the application identifies the 
utterances "Press one for date" and "Press two for 
money" IVR system 10 sends a prompt signal shown on 
successive lines of Fig. 7 as: DTMF signals representing 
"p" and "r" ; the actual utterance "Press one for date" ; 
DTMF signals representing "t" and "e" ; DTMF signals 
representing "p" and "r"; the actual utterance "Press 
two for money" ; and DTMF signals representing "e" and 
"y" . This composite prompt signal including two 
utterances and 24 DTMF signals representing content of 
the utterance in coded form (identified in Fig. 7 as 
prompt signals "A") is received and stored by ACG unit 
16 . 

Continuing in Fig. 7, after ACG unit 16 provides a 
simulated user response in the form of a touchtone 
signal for the number "1", IVR system 10 provides prompt 
signals "B" as illustrated, including one utterance 
"Enter date" and 12 DTMF signals representing the 
letters "e" , "n" and "t" , "e" . This is repeated for 
prompt signals "C" , which represents repetition back to 
the user of the user's entry of the touchtone keyed date 
"11 22 67 " . Prompt signals "B" ^and "C"_ are s toured by ^CG 
unit 16 . 
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As previously described, data representative of 
content of correct utterances which IVR system 10 is 
intended to provide in response to the simulated user 
inputs provided by ACG unit 16 has been stored. This 
5 may be accomplished, for example, by storing the ASCII 

representation of the utterance label or, alternatively, 
by storing the DTMF representation of such label. In 
either case, decode circuit 34 of ACG unit 16 is 
effective to identify the representation of the 

10 utterance label as included in received prompt signals. 
Comparison circuit 36 is responsive to the received 
representations of the utterance label for purposes of 
comparison with previously stored correct utterance 
label representation. Comparisons can be carried out in 

15 any suitable manner, for example, by comparing received 
DTMF signals to stored DTMF signals, or comparing 
received ASCII characters to stored ASCII characters. 
Comparisons may also be carried out in real time as 
prompt signals are received, or carried out subsequently 

20 based on storage of utterance label representations as 
received, with comparison performed later. 

By comparing content of an utterance label as 
represented by coded signals included in a prompt signal 
with content of a correct utterance label (i.e., 

25 accurately responsive to a particular simulated user 
input) as represented by previously stored data, 
discrepancies can be identified. Such discrepancies may 
include an inaccurate utterance, a missing utterance, an 
out of order utterance, etc. Absent any discrepancy, 

30 accuracy of call -flow may be confirmed. In this manner, 
call-flow is verified for a single simulated user call. 
By providing ACG units 16, 17, 18 of suitable capacity, 

cal l-fl ow ve rif ication may be car ried out by a 

simultaneous and continuing volume of calls of identical 
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or differing content to provide verification under high 
volume or saturation conditions. Thus, any level of 
incoming user calls can be simulated. Call -flow 
verification can proceed on a call-by-call or other 
basis, without intrusion on routine calls as may be 
placed by users. 

Fig. 8 is a flow chart illustrating an example of a 
call-flow verification method. 

At 40, an IVR system (or more generally, an 
interactive audio system) adapted to operate in a CFV 
mode is provided. As described, encoding circuit 30 and 
activation circuit 32, which may be implemented via 
software, hardware, or a combination thereof, are 
provided within IVR system 10 for this purpose. 

At 41, predetermined prompt data representative of 
content of correct utterances to be provided by the IVR 
system in response to specific data inputs in the course 
of a simulated user call is stored in ACG unit 16, so as 
to be available for purposes of comparison. 

At 42, the CFV mode of the IVR system is activated 
by sending the appropriate CFV sequence code. This can 
be accomplished via a call placed to the IVR system, via 
a direct connection, via a switch on the IVR system, 
etc . 

At 43, a simulated user call is placed to the IVR 
system (e.g., via suitable public or private network). 
Alternatively, this may be accomplished via continuation 
of a call placed for purposes of Step 42. 

At 44, via the call connection established in Step 
43, an initial (e.g., standard greeting/opening 
utterance) prompt signal is received from the IVR 
isystem. 

At 45, via the same call connection, a first 
simulated user data input (e.g., one or more keypad key 
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strokes) is sent to th e IVR SY^.^ejTLJL n res ponse to the 
init ial pr ompt signal. 

At 46, via the same call connection, a second 
prompt signal responsive to the first data input is 
5 received from the IVR system and data representative of 
utterance content is stored so as to be available for 
comparison purposes. Alternatively, comparisons may be 
carried out as data is received (e.g., on a real time 
basis) . 

10 At 47, via the same call connection, Steps 45 and 

46 are repeated for successive simulated user data 
inputs and responsive prompt signals received from the 
IVR system, as appropriate and desired. 

At 48, the content of each utterance label (as 

15 represented by coded signals included in respective 
prompt signals) is compared with content of a model 
utterance label (as represented by predetermined prompt 
data stored in Step 41) and discrepancies identified. 
The "content" of an utterance label for this purpose 

20 may, for example, be the first two and last two 

characters of the actual utterance label, all characters 
of the actual utterance label, or some other appropriate 
content, as previously described. 

At 49, a record of discrepancies identified at Step 

25 48 and/or lack thereof is provided in memory, as a 

printed report, as an accessible file, etc., for use in 
reviewing the results of the call -flow verification as 
performed. 

Variations of the above will be apparent to skilled 
30 persons. For example, once preparatory steps (e.g., 

Steps 40 and 41) have been initially implemented for a 
particular IVR system, subsequent call-flow verification 
for that sy stem can^e carried out w ithout the n eed to 
repeat such steps. Also, if the CFV mode has been 
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previously activated and remains activated, further 
call-flow verification can be carried out without 
performance of Step 42. 

Fig. 9 is a representation of an example of a CFV 
sequence code for use, as described, to activate and 
deactivate the CFV mode on a per port (i.e., per line) 
basis in an interactive audio system, such as IVR system 
10. The CFV sequence code may be transmitted in the 
form^^of_a series,. o f DT MF sigrijal^ sent to IVR system 10 
via a public telephone system or other means. In a 
currently preferred configuration a CFV sequence code 
includes at least four digits. As represented in the 
Fig. 9 example, digits 1 and 2 are identification (ID) 
digits used to indicate that the CFV mode is to be 
activated. Upon transmission of the CFV sequence code 
to IVR system 10 at a time when its CFV mode is already 
activated, ID digits 1 and 2 indicate that the CFV mode 
is to be deactivated. ID digits may have any suitable 
values, which are programmed into IVR system 10 and then 
transmitted by ACG unit 16 to activate/deactivate the 
CFV mode for call-flow verification purposes. 

The third digit of the CFV sequence code of Fig. 9 
is the "framing" digit. A code protocol may, for 
example, use the symbol to indicate the actual 

utterance is to be included and framed on each side by 
DTMF signals representing its content, as previously 
described. Inclusion of any other letter or number at 
the digit 3 position would then indicate that the actual 
utterance is to be excluded, so that the prompt signals 
will include DTMF signals representative of utterance 
content, but not the utterances themselves. 

The fourth digit of the CFV sequence code, as 
illustrated, is the "extent" digit. The code protocol 
may, for example, use the symbol "0" or zero to indicate 
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that all characters of each utterance label are to be 
transmitted in DTMF format (see Fig. 6 and description 
thereof) . Inclusion of any one of numbers 1 through 9 
at the digit 4 position would then indicate that that is 
the number of characters of each utterance label which 
is to be transmitted in DTMF format (see Figs. 4 and 5) . 

The presently preferred CFV sequence code thus 
includes two digits to signify that PVM mode is to be 
activated or deactivated and single digits to 
respectively signify the framing and extent of utterance 
content to be encoded- In Fig. 9 a particular sequence 
and number of these digits is illustrated. In 
application by skilled persons it will be apparent that 
different coding protocols may be employed depending on 
the protocols deployed in a telecommunications system. 
These protocols can then be used to relay information as 
to activation, framing and extent provided to the IVR 
system by network, direct connection, manipulation of 
switches at the IVR system itself, etc. For example, 
varying DTMF formats are present for telephony systems 
such as Autovon terminals. Government Autovon 
(Automatic Voice Network) telephones have 16 DTMF 
combinations, the extra four being used for 
"precedence", which is a designation assigned to a phone 
call by the caller to indicate to communications 
personnel the relative urgency (therefore the order of 
handling) of the call and to the called person the order 
in which the message is to be noted. Similarly, DTMF to 
dial pulse code can be utilized in a PBX system, through 
DTMF to dial pulse conversion in which the PBX converts 
DTMF to accomodate dialing to rotary dialing. 

With an understanding of the invention, it will be 
apparent that further benefits can be provided in 
verifying operation of IVR systems, as well as other 
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types of interactive audio systems. For example, 
detection of "talk off", occurring when an IVR system is 
subject to false user data inputs (i.e., as a result of 
the presence of signals appearing to be DTMF^ signals 
5 included within spoken prompts) during a call, can be 
determined using discrepancy analysis of stored call- 
flow verification results. 

While there have been described the currently 
preferred embodiments of the invention, those skilled in 
10 the art will recognize that other modifications may be 
made without departing from the invention and it is 
intended to claim all modifications and variations as 
^ fall within the scope of the invention. 
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